Second Order Approximations for Probability Models

In this paper, we derive a second order mean field theory for directed graphical probability models. By using an information theoretic argument it is shown how this can be done in the absense of a partition function. This method is the direct generalisation of the well-known TAP approximation for Boltzmann Machines. In a numerical example, it is shown that the method greatly improves the first order mean field approximation. The computational complexity of the first (second) order method is linear (quadratic) in the network size and is exponential in the potential size. For a restricted class of graphical models, so-called single overlap graphs, the second order method has comparable complexity to the first order method.